Method and system for sorting and identifying medication via its label and/or package

ABSTRACT

Disclosed herein is an improved pharmaceutical management system and methods implemented by the system for sorting and identifying a medicine via its label and/or package. The method comprises steps of: (1) receiving a plurality of raw images of a package of a medication; (b) juxtaposing two of the plurality of raw images to produce a combined image, in which the two raw images are different from each other; (c) processing the combined image to produce a reference image; and (d) establishing the medication library with the aid of the reference image. The system comprises an image capturing device, an image processor, and a machine learning processor. The image processor is programmed with instructions to execute the method for producing a combined image.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present disclosure in general relates to the field of pharmaceuticalmanagement systems. More particularly, the present disclosure relates tomethod and system for sorting and identifying a medicine via its labeland/or package.

2. Description of Related Art

An efficient storage and managing system and/or method of medicalsupplies and pharmaceutical products is an important prerequisite forthe smooth operation of a hospital system, as well as for providing highquality patient care. In such case, accuracy for dispensing aprescription is of paramount importance for all medical institutions.Though automated dispensing cabinets (ADCs) have been introduced intohospitals for over 20 years, yet, pharmacy errors still exist. Asadministering medication by ADCs relies heavily on precise adherence tostandard protocols by clinical staff, thus it is impossible tocompletely eliminate the final human errors. For instance, pharmacy maystock the wrong medication in a given drug cabinet, or a clinician maypick a “look-alike” medication from an adjacent drug drawer. Therefore,identifying medications via their appearances with human eyes is notreliable, nor can it be applied to medication management systems.

Deep learning methods is part of a broader family of machine learningmethods based on learning data representations with multiple levels,which are obtained by composing simple but not-linear modules that eachtransforms the representation at one level into a representation at ahigher, slightly more abstract level. With the composition of enoughsuch transformations, complex functions such as classification tasks canbe learned. Therefore, deep learning is making major advances in solvingproblems existing in the artificial intelligence community for manyyears and are being applied to many technical fields. As forpharmaceutical management, the superior performance of deep learning inidentifying appearances of objects seems to be a promising solution forthe shortcomings of current drug dispensing technologies. However,considering massive types of medicine packages with certain appearancesimilarity involved, simply practicing deep learning failed to producedesirable outcome.

In view of the foregoing, there exists in the related art a need for animproved method and system for medication management (e.g., sorting andidentifying medications via their labels and/or package).

SUMMARY

The following presents a simplified summary of the disclosure in orderto provide a basic understanding to the reader. This summary is not anextensive overview of the disclosure and it does not identifykey/critical elements of the present invention or delineate the scope ofthe present invention. Its sole purpose is to present some conceptsdisclosed herein in a simplified form as a prelude to the more detaileddescription that is presented later.

As embodied and broadly described herein, the purpose of the presentdisclosure is to provide an improved pharmaceutical management systemand methods implemented by the system for identifying clinical drugs,such that the efficiency and accuracy in dispensation of medication canbe highly improved.

In one aspect, the present disclosure is directed to a computerimplemented method for building a medication library. In someembodiments, the method comprises: (a) receiving a plurality of rawimages of a package of a medication; (b) juxtaposing two of theplurality of raw images to produce a combined image, in which the twojuxtaposed raw images are different from each other; (c) processing thecombined image to produce a reference image; and (d) establishing themedication library with the aid of the reference image.

In some optional embodiments, the method further comprises a step ofcapturing the plurality of raw images of the package of the medicationsimultaneously prior to the step (a).

According to certain embodiments of the present disclosure, preferably,the medication is in the form of a blister package.

According to certain embodiments of the present disclosure, the step (b)comprises: (b-1) respectively processing the plurality of raw images toproduce a plurality of first processed images respectively havingdefined contours; (b-2) identifying the corners of each defined contoursof the first processed images to determine the coordinates thereof;(b-3) rotating each of the first processed images of the step (b-1)based on the determined coordinates of the step (b-2) to produce aplurality of second processed images; and (b-4) combining any two of thesecond processed images to produce the combined image of the step (b).

According to certain embodiments of the present disclosure, the each ofthe plurality of raw images of step (b-1) are subjected to treatments of(i) a grayscale-converting treatment, (ii) a noise-reduction treatment,(iii) an edge-identification treatment, (iv) a convex hull treatment,and (v) a contouring treatment.

According to embodiments of the present disclosure, each treatments (i)to (v) may be performed independently in any sequence; and in eachtreatments (i) to (v), the raw images of the step (b-1) or imagestreated by any of said treatments (i) to (v) other than the currenttreatment are used.

In some embodiments of the present disclosure, the step (b-2) is carriedout by a line transforming algorithm or a centroid algorithm.

Preferably, the aforementioned combined image has the image of bothsides of the blister package of the medication.

According to some embodiments of the present disclosure, the step (c) iscarried out by a machine learning algorithm.

In another aspect, the present disclosure pertains to a computerimplemented method for identifying a medication via its blister package.The method comprises: (a) obtaining the front and back images of theblister package of the medication simultaneously; (b) juxtaposing thefront and back images of the step (a) to produce a candidate image; (c)comparing the candidate image with a reference image of a medicationlibrary established by the aforementioned method; and (d) outputting theresult of the step (c).

According to certain embodiments of the present disclosure, the step (b)comprises: (b-1) respectively processing the front and back images toproduce two first processed images respectively having defined contours;(b-2) identifying the corners of each defined contours of the two firstprocessed images to determine the coordinates thereof; (b-3) rotatingeach of the two first processed images of the step (b-1) based on thedetermined coordinates of the step (b-2) to produce two second processedimages; and (b-4) combining the two second processed images of the step(b-3) to produce the candidate image of the step (b).

According to certain embodiments of the present disclosure, the frontand back images of step (b-1) are respectively subjected to treatmentsof: (i) a grayscale-converting treatment, (ii) a noise-reductiontreatment, (iii) an edge-identification treatment, (iv) a convex hulltreatment, and (v) a contouring treatment.

In some embodiments, each treatments (i) to (v) may be performedindependently in any sequence; and in each treatments (i) to (v), thefront and back images of the step (b-1) or images treated by any of saidtreatments (i) to (v) other than the current treatment are used.

In some optional embodiments, the step (b-2) is carried out by a linetransforming algorithm or a centroid algorithm.

In some optional embodiments, the step (d) is carried out by a machinelearning algorithm.

In some optional embodiments, the method further comprises a step oftransferring the candidate image into the medication library prior tothe step (c).

In yet another aspect, the present disclosure is directed to apharmaceutical management system, which includes an image capturingdevice, an image processor, and a machine learning processor, configuredto implement the present method.

More specifically, the image capturing device is configured to capture aplurality of images of a package of a medication. The image processor isprogrammed with instructions to execute a method for producing acandidate image, in which the method comprises: (1) respectivelyprocessing the plurality of images of the package of the medication toproduce a plurality of first processed images respectively havingdefined contours; (2) identifying the corners of each defined contoursof the first processed images to determine the coordinates thereof; (3)rotating each of the first processed images based on the identifiedcoordinates of the step (2) to produce a plurality of second processedimages; and (4) juxtaposing two of the second processed images of thestep (3) to produce the candidate image, in which the two secondprocessed images are different from each other. In addition, the machinelearning processor is programmed with instructions to execute a methodfor comparing the candidate image with a reference image of theaforementioned medication library. The result produced by the machinelearning processor may subsequently be outputted to notify an operator,whom is dispensing the medication.

In some embodiment of the present disclosure, the image capturing devicecomprises a transparent board on which the medication is placed and twoimage-capturing units individually disposed above each side of thetransparent board.

In some embodiment of the present disclosure, each of the plurality ofimages of the step (1) are subjected to treatments of: (i) agrayscale-converting treatment, (ii) a noise-reduction treatment, (iii)an edge-identification treatment, (iv) a convex hull treatment, and (v)a contouring treatment.

In some embodiments, each treatments (i) to (v) may be performedindependently in any sequence; and in each treatments (i) to (v), theplurality of images of the step (1) or images treated by any of saidtreatments (i) to (v) other than the current treatment are used.

Additionally or alternatively, said step (2) may be carried out by aline transforming algorithm or a centroid algorithm.

In some optional embodiments, the method for comparing the candidateimage with a reference image of the aforementioned medication library iscarried out by a machine learning algorithm.

By virtue of the above configuration, the medication management methodand system for sorting and identifying can be executed in a real-timemanner, thereby shortening the whole processing time of imageclassification regardless of the orientation of medications during drugsdispensation.

Furthermore, the accuracy of drug identification via their appearanceand/or blister packages can be increased, and man-made mistakes duringmedication dispensing can be greatly reduced. Therefore, medication-usesafety could be improved.

Many of the attendant features and advantages of the present disclosurewill becomes better understood with reference to the following detaileddescription considered in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The present description will be better understood from the followingdetailed description read in light of the accompanying drawings, where:

FIG. 1 is a flow chart illustrating a method 100 according to oneembodiment of the present disclosure;

FIG. 2 is flow chart illustrating a method 200 according to oneembodiment of the present disclosure;

FIG. 3 is a diagram illustrating a pharmaceutical management system 300according to another embodiment of the present disclosure; and

FIG. 4 are schematic drawings depicting how the corners of a package areidentified via use of a centroid algorithm according to one example ofthe present disclosure.

In accordance with common practice, the various describedfeatures/elements are not drawn to scale but instead are drawn to bestillustrate specific features/elements relevant to the present invention.Also, like reference numerals and designations in the various drawingsare used to indicate like elements/parts.

DETAILED DESCRIPTION OF THE INVENTION

The detailed description provided below in connection with the appendeddrawings is intended as a description of the present examples and is notintended to represent the only forms in which the present example may beconstructed or utilized. The description sets forth the functions of theexample and the sequence of steps for constructing and operating theexample. However, the same or equivalent functions and sequences may beaccomplished by different examples.

I. Definition

For convenience, certain terms employed in the specification, examplesand appended claims are collected here. Unless otherwise defined herein,scientific and technical terminologies employed in the presentdisclosure shall have the meanings that are commonly understood and usedby one of ordinary skill in the art. Also, unless otherwise required bycontext, it will be understood that singular terms shall include pluralforms of the same and plural terms shall include the singular.Specifically, as used herein and in the claims, the singular forms “a”and “an” include the plural reference unless the context clearlyindicates otherwise. Also, as used herein and in the claims, the terms“at least one” and “one or more” have the same meaning and include one,two, three, or more.

Notwithstanding that the numerical ranges and parameters setting forththe broad scope of the invention are approximations, the numericalvalues set forth in the specific examples are reported as precisely aspossible. Any numerical value, however, inherently contains certainerrors necessarily resulting from the standard deviation found in therespective testing measurements. Also, as used herein, the term “about”generally means within 10%, 5%, 1%, or 0.5% of a given value or range.Alternatively, the term “about” means within an acceptable standarderror of the mean when considered by one of ordinary skill in the art.Other than in the operating/working examples, or unless otherwiseexpressly specified, all of the numerical ranges, amounts, values andpercentages such as those for quantities of materials, durations oftimes, temperatures, operating conditions, ratios of amounts, and thelikes thereof disclosed herein should be understood as modified in allinstances by the term “about”. Accordingly, unless indicated to thecontrary, the numerical parameters set forth in the present disclosureand attached claims are approximations that can vary as desired. At thevery least, each numerical parameter should at least be construed inlight of the number of reported significant digits and by applyingordinary rounding techniques. Ranges can be expressed herein as from oneendpoint to another endpoint or between two endpoints. All rangesdisclosed herein are inclusive of the endpoints, unless specifiedotherwise.

As used herein, the term “blister pack” or “blister package” encompassesany type of layered package comprising a product contained betweensheets of material; wherein the sheets are adhered or sealed together bymethods known to those skilled in the art; for example, the sheets aresealed by heat and/or pressure activated adhesive. The sheets ofmaterial are commercially available as individual sheets (for handpackaging) or, preferably, as a continuous web sheet on roll stock (formachine packaging). The primary component of a blister package is acavity or pocket made from a formable web, usually a thermoformedplastic. The cavity or pocket is large enough to contain the good whichis housed in the blister package. Depending on the application, ablister pack may have a backing of thermoformable material. Forpharmaceutical fields, “blister packages” are commonly used as unit-dosepackaging for pharmaceutical tablet, and contain drug informationprinted on the back thereof. In addition, the sheets are available in avariety of thicknesses.

II. Description of the Invention

What is most concerned in patient care is medication-use safety. Fillingprescription and dispensing medicines correctly is of paramountimportance for patient care. Since the conventional automated dispensingsystems, such as ADCs, still has room for improvement, this inventionaims to provide an improved, highlighted deep learning method thataddresses the afore-mentioned issue. Furthermore, the present inventionalso aims to develop an automatic medication verification (AMV)apparatus in a real-time operating system so as to reduce the workloadof clinical stuff.

More specifically, the “highlighted” deep learning method refers to amethod of processing and highlighting features or images beforeinputting them into algorithms, such that the feature information can belearned in a more emphasized manner. Generally, in order to betterdivide the identified targets and to expose inherent descriptivefeatures, the feature processing is achieved by various image-processingsteps which lead to conjugate both cropped pictures of blister packageappearance into a fixed signature template, thereby facilitating thesubsequent classification process in learning networks.

1. Method for Building Mediation Library

The first aspect of the present disclosure is directed to a method forbuilding a medication library implemented in a computer-readable storagemedium. References are made to FIG. 1.

Referring to FIG. 1, which is a flow chart of a method 100 implementedon a computer according to one embodiment of the present disclosure. Themethod comprises at least the following steps,

(S110) receiving a plurality of raw images of a package of a medication;

(S120) juxtaposing two of the plurality of raw images to produce acombined image, in in which the two raw images are different from eachother;

(S130) processing the combined image to produce a reference image; and

(S140) establishing the medication library with the aid of the referenceimage.

Before the method 100 is implemented, a plurality of raw images of amedication package (e.g., a blister package) are captured by anywell-known manners, such as an image capturing devices (e.g., videocamera). Then, in the step S110 of the present method, the capturedimages are automatically forwarded to a device and/or system (e.g., acomputer) having instructions embedded thereon for executing the presentmethod 100. In the following step S120, two of the plurality of rawimages received by such device and/or system are juxtaposed to eachother to produce a combined image. It is worth noting that the twojuxtaposed raw images are different from each other. In one exemplaryembodiment, the medication is in the form of a blister package; hence,the plurality of raw images taken by the image capturing device mayencompass both sides of the blister package. In other words, two imagesrespectively showing the front and back of the blister package arejuxtaposed to each other to produce a combined image. In order toimprove the accuracy of appearance recognition for the subsequentmachine learning process, the raw images are highlighted and processedto give a neat image having a pre-determined feature (e.g., in apre-determined pixels size, etc.), in which all background portionsother than the main object (i.e., the image of blister package and/orlabels) is completely removed.

In the step S130, the combined images are then used to train a machinelearning algorithm embedded in the computer (e.g., a processor) toproduce a reference image. The steps S110 to S130 may be repeated manytimes, each time using a medication package different from the onebefore. Eventually, a medication library can be established with the aidof the reference image (the step S140).

Returning to the step S120, which generally includes the followingsteps,

(S121) respectively processing the plurality of raw images to produce aplurality of first processed images respectively having definedcontours;

(S122) identifying the corners of each defined contours of the firstprocessed images to determine the coordinates thereof;

(S123) rotating each of the first processed images of the step S121based on the determined coordinates of the step S122 to produce aplurality of second processed images; and

(S124) combining any two of the second processed images to produce thecombined image of the step S120.

In the step S121, the plurality of raw images are respectively processedby an image processor to eventually produce a plurality of firstprocessed images respectively having well defined contours. The imageprocessor employs several algorithms to eliminating background noisesand extracting target features from the images. Specifically, each ofthe plurality of raw images of the step S121 are respectively subjectedto treatments of: (i) a grayscale-converting treatment, (ii) anoise-reduction treatment, (iii) an edge-identification treatment, (iv)a convex hull treatment, and (v) a contouring treatment. It is worthnoting that each treatments (i) to (v) are performed independently inany sequence.

More specifically, (i) the grayscale-converting treatment is executed byuse of a color-converting algorithm to change BGR colors into gray-scale(S1211); (ii) the noise-reduction treatment is executed by use of afilter algorithm to minimize the background noise (S1212); (iii) theedge-identification treatment is executed by use of an edge identifyingalgorithm to determine the coordinates of each edges of the blisterpackage in the image (S1213); (iv) the convex hull treatment is executedby use of convex hull algorithm to calculate the actual area of theblister package of the medication in the image (S1214); and (v) thecontouring treatment is executed by use of a contour defining algorithmto extract and highlight the main portion of the blister package(S1215).

According to certain embodiments, the raw images in the step S121 aresubjected to all of the treatments (i) to (v) before proceeding to thestep S122. As mentioned before, these treatments may be performedindependently in any order, hence except the raw images form the stepS121, images derived from one treatment can be the raw images in thenext treatment. For example, the raw images of the step S121 are firstsubjected to the treatment (i) or step S1211, in which BGR colors of theraw image are converted to grayscale, thereby generating a greyscaleimage. This grayscale image derived from the treatment (i) may continueto be treated with any of treatments (ii) to (v), until all treatments(i) to (v) have been successfully applied thereon, thereby generating afirst processed image having defined contours.

Alternatively, or optionally, the raw images of the step S121 are firstsubjected to treatment (iv) or step S1214, in which a convex hullalgorithm is applied thereon. The convex hull algorithm is to find theconvex hull of a finite set of points in the plane or otherlow-dimensional Euclidean spaces. In the present disclosure, the convexhull is defined by calculating the actual area of the blister package ofthe medication in the image. The image derived from treatment (iv) orstep S1214 may then be treated with any treatments of (i), (ii), (iii)or (v), until all treatments (i) to (v) have been successfully appliedthereon, thereby generating a first processed image having definedcontours.

It should be noted that the afore-indicated treatments (i) to (v) orsteps S1211 to S1215 can be performed sequentially, randomly, and/orrepeatedly (e.g., the treatment (iii) can be repeated one or moretimes); preferably, they are performed in the order of (i) to (v) orfrom the steps S1211 to S1215.

By performing the afore-mentioned steps S1211 to S1215, a plurality offirst processed images, each having defined contours, are generated fromthe plurality of raw images, in which the background noise in the rawimages is eliminated and the main portion of the blister package isextracted and highlighted for further image processing procedure.

Proceed to the step S122, in which the corners in each of the firstprocessed images are identified FIG. 1). In this step, the coordinatesof at least one corner of each of the first processed images aredetermined regardless of the profile shapes of the blister packages.Preferably, the coordinates of four corners of each of the firstprocessed image derived from the step S121 are identified. The goal ofthis step is to project at least three straight lines from the profileedge of the blister package, then derive the closest quadrilateral shapeand the corner coordinates thereof by geometric inference. As theblister package might come in various shapes and/or profiles, thusdifferent algorithms may be adopted to identify the corner(s) from thefirst processed images of a medication package. For instance, if thecontour shape of the medication package in the first processed image isregular quadrilateral (e.g., a rectangle), a line transforming algorithmis applied to identify the four corners. On the other hand, if themedication package in the first processed image is in irregularquadrilateral shape, such as the atypical polygon having three straightedges and one curved edge, then a centroid algorithm is applied forcorner identification. By virtue of the above configuration, thecoordinates of each corner can be determined regardless of theorientation and shapes of the medication packages.

Identifying the corners of each processed images in the step S122 notonly serves the purpose of setting up anchor points for the execution ofthe subsequent rotating step (S123) and combining step (S124), but alsothe purpose of ensuring that the entire package is encompassed in thefirst processed image for further analysis. It is worth noting that thefour corners of the blister package images determined above have to bearranged in clockwise or anticlockwise manner, such that the firstprocessed images can be rotated into a predetermined position based onthe determined coordinates of the four corners (S123). The predeterminedposition can vary based on practical requirements. In some instances,the predetermined position refers to the position in which the short andthe long sides of the blister packages are respectively parallel withthe X and Y axes in a Cartesian coordinate system. Preferably, theblister package is oriented in the manner that the image only needs tobe rotated once to have its short and long sides respectively beingparallel to the X and Y axes in a Cartesian coordinate system. Inpractice, several perspective transformation algorithms may be used torotate the first processed images, which was set to have apre-determined pixel sizes (e.g., 448×224 pixels), thereby produces thesecond processed (rotated) images with the predetermined position.

Next, in the step S124, any two of the second processed images of amedication are placed side by side (or juxtaposed to each other) andthereby produces a combined image. It is worth noting that the combinedimage obtained in this step can include various permutations, which isbeneficial for building up the library database as much as possible. Twoof the second processed images may be the opposite sides of the blisterpackage regardless the fact that it has been inverted or not.Preferably, two of the second processed images respectively showing eachsides of the blister package of the medication having the sameorientation are selected and juxtaposed to each other to produce acombined image that encompasses maximum feature information of thepackage therein.

In the steps S130 and S140, the combined image is used to train amachine learning algorithm embedded in the computer to produce areference image in the purpose of establishing the medication library.The combined image, which includes the medication information, isclassified with its features stored as a reference information in themedication library and may be retrieved in the future for identifying acandidate package. In some embodiments, at least one combined image isinputted into the machine learning algorithm. In the exemplaryembodiment, more than 10 to 20,000 combined images, such as 10, 100,200, 300, 400, 500, 1000, 1,100, 1,200, 1,300, 1,400, 1,500, 2,000,2,500, 3,000, 3,500, 4,000, 4,500, 5,000, 5,500, 10,000, 11,000, 12,000,13,000, 14,000, 15,000, 16,000, 17,000, 18,000, 19,000 and 20,000combined images are inputted to train the machine learning algorithm toestablish the medication library. Every image can be used to train themachine learning system to transform the image information into thereference information, and the reference information is subsequentlystored in the built-in library in the device and/or system.

It is worth noting that the machine learning programming system suitablefor use in the present disclosure can be any well-known vision objectdetection models, which include but are not limited to, deformable partsmodel (DPM), region convolutional neutral network (R-CNN), Fast R-CNN,Faster R-CNN, Mask R-CNN and YOLO, with or without being optimized atcertain criteria according to practical needs. Preferably, the visionobject detection models suitable for training the learning step of thepresent disclosure are optimized, at least in parameters like pixels ofthe input images, numbers and sizes of bounding boxes, and anchor boxes.According to the present disclosure, the combined image processed by theabove-mentioned steps and then inputted into the learning system must be“full bleed image” and in a pre-determined size (e.g., with fixedpixels). Therefore, the number and sizes of anchor boxes for operationcan be minimized (e.g., minimized to only one anchor box) to enhance thecomputing speed and efficiency. In other words, by virtue of theafore-mentioned step for juxtaposing two images into one, the machinelearning system can run smoothly and faster even when processing hugevolumes of data of massive types of medicine packages. Furthermore, theentire processing time is shortened, which further enhance the efficacyin producing a medication library.

By performing the aforementioned steps S130 and S140, the trainingresults for each distinct medication can be outputted as aclassification model when the medication library is in practice.

2. Method for Identifying a Medication

A second aspect of the present disclosure is directed to a method foridentifying a medication via its package (e.g., a blister package).References are made to FIG. 2.

Referring to FIG. 2, which is a flow chart illustrating a method 200according to one embodiment of the present disclosure. The method 200comprises steps of,

(S210) obtaining the front and back images of the blister package of themedication simultaneously;

(S220) juxtaposing the front and back images of the step S210 to producea candidate image;

(S230) optionally transferring the candidate image into the medicationlibrary;

(S240) comparing the candidate image with a reference image of amedication library established by the above-mentioned method; and

(S250) outputting the result of the step S240.

Preferably, the method 200 can be implemented via a non-transitoryprocessor-readable storage medium embedded with instructions forexecuting steps of the present method, as well as a medication library.The medication library can be originally stored and built-in or can beestablished by the afore-mentioned method 100.

In the image-receiving step (S210), two raw images of a medication arecaptured simultaneously. The two raw images are, preferably,individually each side of the blister package of the medication (i.e.,the front and back images). The raw images can be obtained by anywell-known manner and preferably by at least one image capturing devices(e.g., video camera). In one exemplary embodiment, the two raw images ofthe blister package are captured simultaneously by two image capturingdevices disposed over two sides of the medication (e.g., the front andback of the medication); hence, the two raw images will encompassmedication information on both sides of the blister package. To improvethe accuracy of appearance recognition for the subsequent machinelearning process, the two raw images obtained in the step S210 arejuxtaposed right next to each other to produce a candidate image (thestep S220). Like the method 100, the goal is to produce a neat imagehaving a pre-determined feature (e.g., front and back images disposedside-by-side, in a pre-determined pixels size and etc.) withoutinformation other than the main object (i.e., only with the image ofblister package and/or labels).

The candidate image produced in the step S220 is then compared withreference images stored in the medication library (S240), and the result(i.e., drug information for the certain medication) is outputted to auser (S250). The step S220 may be performed at the same or differentcomputing device for executing the steps S210, S240 and S250. Accordingto optional embodiments of the present disclosure, the step S220 isexecuted at a computing device different from the one for executing thesteps S210, S240 and S250, accordingly, an optional step S230 isperformed, in which the processed image or the candidate image producedin the step S220 is transferred to the computing device where the stepsS240 and S250 are to be performed.

Returning to the step S220, which is similar to the step S120 of themethod 100, and generally includes steps of

(S221) processing the front and back images to produce two firstprocessed images respectively having defined contours;

(S222) identifying the corners of each defined contours of the two firstprocessed images to determine the coordinates thereof;

(S223) rotating each of the two first processed images of the step S221based on the determined coordinates of the step S222 to produce twosecond processed images; and

(S224) combining the two second processed images of the step S223 toproduce the candidate image.

In the step S221, the front and back images are respectively processedby an image processor to produce a plurality of first processed images,in which each processed image has a well-defined contour. To thispurpose, the front and back images of the step S221 are respectivelysubjected to treatments of: (i) a grayscale-converting treatment, (ii) anoise-reduction treatment, (iii) an edge-identification treatment, (iv)a convex hull treatment, and (v) a contouring treatment. It is worthnoting that treatments (i) to (v) are performed independently in anysequence. In practice, several well-known algorithms may be used toeliminating background noises and extracting target features from thetwo raw images.

More specifically, (i) the grayscale-converting treatment is executed byuse of a color-converting algorithm (S2211); (ii) the noise-reductiontreatment is executed by use of a filter algorithm to minimize thebackground noise (S2212); (iii) the edge-identification treatment isexecuted by use of an edge identifying algorithm to determine thecoordinates of each edges of the blister package in the image (S2213);(iv) the convex hull treatment is executed by use of convex hullalgorithm to calculate the actual area of the blister package of themedication in the image (S2214); and (v) the contouring treatment isexecuted by use of a contour defining algorithm to extract and highlightthe main portion of the blister package (S2215).

In practice, the raw images in the step S221 are subjected to alltreatments (i) to (v) before the method proceeding to the step S222. Asmentioned before, these treatments can be performed in any order, hencethe raw images obtained in the step S221 or the image derived from onetreatment are used in the next treatment. For example, the raw images ofthe step S221 are first subjected to the treatment (i) or step S2211, inwhich BGR colors of the raw image are converted to grayscale, therebygenerating a greyscale image. This grayscale image derived from thetreatment (i) may continue to be treated with any of treatments (ii) to(v), until all treatments (i) to (v) have been successfully appliedthereon, thereby generating a first processed image having definedcontours.

Alternatively, or optionally, the raw images of the step S221 are firstsubjected to treatment (iv) or step S2214, in which a convex hullalgorithm is applied thereon. The convex hull algorithm is to find theconvex hull of a finite set of points in the plane or otherlow-dimensional Euclidean spaces. In the present disclosure, the convexhull is defined by calculating the actual area of the blister package ofthe medication in the image. The image derived from treatment (iv) orstep S2214 may then be treated with any treatments of (i), (ii), (iii),or (v), until all treatments (i) to (v) have been successfully appliedthereon, thereby generating a first processed image having definedcontours.

In one preferable embodiment, treatments are performed sequentially fromstep S2211 to the step S2215. The strategies and algorithms utilized insteps S2211 to S2215 are similar to those described in the step S121,therefore they are not reiterated herein for the sake of brevity.

As would be appreciated, each step of S2212 to S2215 may be executed byany alternative algorithm well-known in the art, as long as thealgorithm achieves the same results described above.

By performing the steps S2211 to S2215, two first processed imagesrespectively having defined contour are generated from the two rawimages (i.e., the front and back images taken from the blister packageof one medicines), in which the background noise in the raw images iseliminated and the main portion of the blister package is extracted andhighlighted for further image processing procedure.

Proceed to the steps S222 to S224, the goals of these steps are the sameas that in steps S122-S124. In the step S222, at least three straightlines of the profile edge of the blister package are determined, thenthe closest quadrilateral shape and the corner coordinates thereof arederived by geometric inference. Once the step S222 for identifyingcorners is accomplished, the step S223 could be performed based on thedetermined coordinates to produce two processed, highlighted images in afixed signature template. In the step S224, the two processed images arejuxtaposed side by side and are combined into one image, so as toproduce a candidate image containing information on both sides of thepackage of the medication. The strategies utilized in the step S220 aresimilar to those described in the method 100, therefore they are notreiterated herein for the sake of brevity.

In the step S240, the combined image or the candidate image is thencompared with reference images stored in the medication library, so asto determine the identity of the medication. In some embodiments, themedication library is stored in the same computing device or in otherprocessor-readable storage medium different from the one havinginstructions for executing the step S220. According to optionalembodiment of the present disclosure, the candidate image is produced ina first processor unit embedded with instructions for executing the stepS220, and the processed image is then transferred to the medicationlibrary (e.g., the medication library built by the method 100 describedabove) located at a second processing unit. The step S240 is executedunder a machine learning instruction embedded in the storage medium tocompare the candidate image with reference images of the medicationlibrary, and the result (either matched or un-matched) is then outputtedto a user. If the comparison indicates that the candidate image matchesor has a highest similarity to a certain reference image in themedication library, then the information of that medicationcorresponding that certain reference image is outputted. If thecomparison fails to produce a matching result between the candidateimage and all reference images in the library, then a notion indicating“no matched result” is outputted.

As mentioned above, the purpose of the machine learning algorithm is toimprove visual recognition of a medication. In some embodiments, theexemplary algorithm can be deep learning algorithm executed under anywell-known vision object detection models with or without beingoptimized at certain criteria according to practical needs. In oneexemplary embodiment, the deep learning is executed under the optimizeddetection model. In another exemplary embodiment, the candidate imageprocessed by the present method must be “full bleed image” and in apre-determined pixels size (e.g., 448×224 pixels); hence, it is possibleto keep the parameters for the operation of machine learning at aminimum level, thereby enhances the computing speed and efficiency.

It is worth noting that, by juxtaposing two processed images into oneimage that termed “combined image”, the present machine learning systemand/or method can run smoothly and faster even when processing hugevolumes of data for massive types of medicine packages. In other words,the processed and highlighted image in fact increases the computingefficiency and accuracy. By virtue of the above technical features, thepresent method 200 aims at identifying a medication via its blisterpackage can enhance the accuracy of medicine identification andsignificantly eliminate human errors during medication dispensing,thereby improves safety in medication usage and the quality in patientcare.

The subject matter described herein could be implemented using anon-transitory, tangible processor-readable storage medium having storedthereon processor-readable instructions that, when executed by theprocessor of a programmable device, control the programmable device toperform a method according to embodiments of the present disclosure.Exemplary processor-readable storage media suitable for implementing thesubject matter described herein include, but are not limited to, RAM,ROM, EPROM, EEPROM, flash memory or other solid state memory technology,CD-ROM, DVD, or other optical storage, magnetic cassettes, magnetictape, magnetic disk storage or other magnetic storage devices, and anyother medium which can be used to store the desired information andwhich can be accessed by the processor. In addition, aprocessor-readable storage medium that implements the subject matterdescribed herein may be located on a single device or computing platformor may be distributed across multiple devices or computing platforms. Insome embodiments, the computing platform is an embedded system withreal-time computing constrains.

3. Pharmaceutical Management System

In another aspect of the subject matter described herein, apharmaceutical management system is provided. Referring to FIG. 3, asystem 300 is depicted, which comprises an image capturing device 310,an image processor 320 and a machine learning processor 330, in whichthe image capturing device 310 and the machine learning processor 330are respectively coupled to the image processor 320. The image capturingdevice 310 is configured to capture a plurality of images of a packageof a medication. In some embodiments, the image capturing device 310comprises in its structure, a transparent board 3101, and two imagecapturing units 3102 individually disposed over each side of thetransparent board, which can be made by glass or acrylate polymers. Inpractical, the medication is placed on the transparent board 3101, andimages of both sides of the medication package are capturedsimultaneously by the two image capturing units 3102. Exemplary, the twoimage capturing units 3102 are real-time digital cameras.

According to the present disclosure, unless otherwise indicated, theimage processor 320 and the machine learning processor 330 respectivelyencompass memories for storing a plurality of instructions that wouldcause the processors to implement the present method(s). In someembodiments, the image processor 320 and the machine learning processor330 are disposed separately as two individual devices; alternatively,they may be disposed in the same hardware. In some embodiments, theimage processor 320 and the machine learning processor 330 arecommunicatively connected to each other. More specifically, the imageprocessor 320 is communicatively connected with the image capturingdevice 310 to receive images captured by the image capturing device 310and configured to execute image processing steps (such as steps S120 andS220) of the present methods thereby producing a candidate image forfurther identification. The machine learning processor 330 iscommunicatively connected with the image processor 320 and configured toimplement image comparison (such as the step S240) of present method formedication identification. The step comprises comparing the candidateimage with a reference image in the medication library established bythe present method. The medication library is communicatively connectedwith the machine learning processor 330. In the exemplary embodimentdepicted in FIG. 3, the present medication library 3301 is stored in themachine learning processor 330. Alternatively or optionally, themedication library may be stored in storage devices connected with themachine learning processor 330 through cable connection or wirelessnetwork.

In some embodiments, the system 300 further comprises a user interface(not shown) configured to output results of medication identification,receive instructions from an external user, and retrieve user inputsback to the image processor 320 and the machine learning processor 330.

The communication among the image capturing device 310, the imageprocessor 320 and the machine learning processor 330 may be embodiedusing various techniques. For instance, the present system 300 maycomprise a network interface to permit communications among the imagecapturing device 310, the image processor 320, and the machine learningprocessor 330 over a network (such as a local area network (LAN), a widearea network (WAN), the Internet, or a wireless network). In anotherexample, the system may have a system bus that couples various systemcomponents including the image capturing device 310 to the imageprocessor 320.

The following Examples are provided to elucidate certain aspects of thepresent invention and to aid those of skilled in the art in practicingthis invention. These Examples are in no way to be considered to limitthe scope of the invention in any manner. Without further elaboration,it is believed that one skilled in the art can, based on the descriptionherein, utilize the present invention to its fullest extent. Allpublications cited herein are hereby incorporated by reference in theirentirety.

Example Example 1 Constructing Medication Library

Preparing Rectified Two-Sided Images (RTIs)

Over 250 medications currently available on the market were collectedfrom the department of hospital pharmacy in Mackay Memorial Hospital(Taipei, Taiwan). Pictures of blister packages of all medications weretaken as many as possible for constructing a database of the medicationlibrary. Images were cropped to minimize background noises and processedto generate multiple combined images (also called “rectified two-sidedimages (RTIs)”) for each medication by utilizing Open Source ComputerVision 2 (OpenCV 2) functions programmed in an image processor. Each RTIwas fitted in a pre-determined template (hereinafter, the rectifiedtwo-sided template, RTT) and encompassed both sides of a blister packageof one medication. Total 18,000 RTIs were obtained for further deeplearning process by utilizing a CNN model.

Strategies for Detecting Corner of Irregular Quadrilateral Shapes ofMedication Packages

In the case that the medication package to be detected has an irregularquadrilateral shape, which is constituted by three straight edges andone curved edges, the centroid algorithm (Table 1) was utilized todetermine the curved edge and the undefined corners by geometricinference.

TABLE 1 The centroid algorithm for corner detection in irregularquadrilateral shape Input: Three edge lines L and the profile shape ofthe blister package blisterContour Output: Four corners P₁, P₂, P₃ andP₄ in cyclic order via lines intersection and geometric inference 1:procedure FINDCORNERS(L, blisterContour) 2: P₁, P₄ ← Two intersectionpoints of three lines in L. 3: M ← The midpoint between two intersectionpoints P₁, P₄. 4: B ← The barycenter of the blister's contourblisterContour. 5: {right arrow over (v)} ← The displacement vector fromM to B. 6: P₂ ← P₁ + 2{right arrow over (v)} 7: P₃ ← P₄ + 2{right arrowover (v)}

Referring to FIG. 4, which exemplarily shows how corner identificationwas performed on a medication package having an irregular quadrilateralshape. As depicted in FIG. 4, a blister package having three straightedges (L₁, L₂, L₃) and one curved edge (C₁) was in a random orientation,in which the intersection of L₁ and L₂ was designated as P₁, and theintersection of L₂ and L₃ was designated as P₄. The goal was todetermine the coordinates of P₂ and P₃, so that the area enclosed by the4 pints (P₁, P₂, P₃ and P₄) and 4 edges (L₁, L₂, L₃ and C₁) wouldencompass the image of the entire package. The mid-point M between P₁and P₄ on the edge L₂ was first determined, and the centroid B of thearea of the blister package was determined by the algorithm cv2.moments(OpenCV 2). The displacement vector v was then calculated based on thecoordinates of the mi-point M and centroid B. Finally, the coordinatesof P₂ and P₃ were determined to be the locations that are respectivelytwo times of the distance v from P₁ and P₄. By virtue of the aboveprocedure, the four points or corners of P₁, P₂, P₃ and P₄ wouldautomatically be oriented in a clockwise or anti-clockwise manner.

Optimizing Machine Learning Ability

In this example, the aim of machine learning was achieved by use ofConvolutional Neural Network (CNN). To this purpose, a tiny versionYOLOv2 (as known as Tiny YOLO) Neural executed by graphics processingunit (GPU) were utilized to train visual recognition for each RTI. Thegeneral concept of the conventional YOLOv2 is to divide one image intomultiple grid cells and uses anchor boxes to predict bounding boxes.Briefly, RTI images, each having 416×416-pixel sizes, were inputted tothe network, in which Tiny YOLOv2 Neural then produced an odd number ofgrid cells in each images; accordingly, there would be only one gridcell in the center (i.e., the center grid cell). Each grid cells and itsneighboring grid cells were then analyzed to determine if they containedany feature information. For example, an input image can be divided into13×13 grid cells, in which bounding boxes with 5 sizes may be predicted,and the confidence score returned by each grid cell is used to determinewhether the grid cell contained the object to be analyzed. In thepresent example, the input images were highlighted and contoured inadvance, thus, the conventional Tiny YOLO was optimized, therebyimproved the identifying efficiency, which led to a reduction in theoperating costs. In detail, since the input image of each RTI wasprocessed to be the “full bleed image” and set to be in a pre-determinedpixels size, the number of anchor boxes for operation was minimized toone, and the size of anchor boxes was set to be 7×7 grid cells.Accordingly, the machine learning network only needed to predict onebounding box with a full size of the input image. The detail parametersfor optimizing Tiny YOLO are listed in Table 2.

TABLE 2 The parameters for the optimized Tiny YOLO network ConventionalTiny Optimized Tiny YOLO network YOLO network Image input size (pixels)416 × 416 224 × 224 Data augmentation Saturation: 50% Saturation: 100%Exposure: 50% Exposure: 100% Bounding boxes for anchor 5  1 Sizes ofbounding boxes 1.08 × 1.19 7 × 7 3.42 × 4.41  6.63 × 11.38 9.42 × 5.1116.62 × 10.52 Jitter parameter 0.2 Off Multi-scale training function OnOff

In practice, the optimized Tiny YOLO was programmed in a machinelearning processor equipped with high resolution graphic card, GEFORCE®GTX 1080 (NVIDIA, USA). The machine learning processor executed the deeplearning process by a training model of the optimized Tiny YOLO. Thetraining criteria for the optimized Tiny YOLO is listed in Table 3.

TABLE 3 The training criteria for the optimized Tiny YOLO Image inputsize (pixels) 224 × 224 Training samples Off augmentation Pre-trainedmodel Non Batch size 8 Maximum epochs 100 epochs (i.e., 168,800 times ofiterative) Trained weights 1 epoch (i.e., 1,688 times of iterative)

Example 2 Evaluating the Medication Sorting Efficiency of aPharmaceutical Management System Implementing Machine Learning Based onthe Medication Library of Example 1

Processed and Combined Images

In order to verify the visual recognition efficiency for the RTIs of thepresent disclosure, both the raw images of medication (non-processed andnon-combined) and the RTIs (processed and combined) were input into thetraining model (the optimized Tiny YOLO) for deep learning. Trainingresults are summarized in Table 4. According to the data in Table 4, the“highlighted” images of the present disclosure was highly efficient intraining visual recognition, as compared to raw images that had not beenprocessed. The higher the F1-score of the training, the higher theefficiency of the deep learning network in the visual recognition ofpackage images. The RTIs that had been processed to the RTT alsosignificantly increased the training efficiency, as compared to thoseimages that had not been processed.

TABLE 4 The comparison between non-processed images and RTIs Experimentgroups Group I Group II Group III Image types Non-processedNon-processed Processed images images images (front side) (back side)(RTIs) Training time (min) 362 462 273 Epochs 65 83 48 Precision (%)77.03 89.39 99.83 Recall 67.44 87.68 99.83 F1-score 65.39 86.48 99.78

Optimized Learning Model

Another comparison was conducted to verify the efficacy of the presenttraining model. Two conventional deep learning models, ResNet 101 andSE-ResNet 101, were utilized to build the training model with the RTIs.The comparison results are summarized in Table 5.

TABLE 5 The comparison between optimized system and conventional systemExperiment groups Group I Group II Group III Training model (algorithm)Optimized Tiny ResNet SE-ResNet YOLO 101 101 Training time (min) 273 9751728 Epochs 48 39 32 Precision (%) 99.83 99.84 99.98 Recall 99.83 99.8399.98 F1-score 99.78 99.79 99.97

The comparing results in tables 4 and 5 demonstrate that the RTIsprocessed by the present method to fit a RTT would increase the trainingefficiency, in particular when the images were processed by theoptimized Tiny YOLO model, in which the training time decreasedsignificantly. All the 18,000 RTIs were inputted into the trainingmodel, such that the medication library of the present disclosure wasbuilt by executing the optimized Tiny YOLO model. The medication libraryand the deep learning network were further stored in an embedded systemfor further application.

Application for Real-Time Medication Identification

In operation, a random medication was chosen and placed in a designedchamber equipped with a transparent board made by glass. Light sourceswere disposed around the transparent board for illumination, and twoBRIO webcams (Logitech, USA) were individually disposed over thetransparent board to ensure the vision field of which encompassed theentire area of the board. On the other hand, the built medicationlibrary had been stored in the real-time embedded computing device,JETSON™ TX2 (NVIDIA, USA) in advance. JETSON™ TX2, which is termed “adeveloper kit,” includes memories, CPU, GPU, USB ports, web antennas andsome other computing elements, thereby enabling image processing andmachine learning steps to be conducted in the same device. In operation,the webcams were connected to JETSON™ TX2 by USB cables, such that theimage of the sides of the chosen medication were simultaneouslytransferred to the processor in real-time. Two raw images of the blisterpackage of the chosen medication were processed to one RTI, afterwardsthe RTI was subjected to the learning model Tiny YOLO programmed inJETSON™ TX2 of which the speed of visual recognition could be up toabout 200 FPS. The time cost for whole procedure (i.e., from imagecapturing to visual identification) is about 6.23 FPS. The outcome ofthe identification will be presented in real-time on an external displaydevice, such as a computer screen or a mobile user interface. Hence, anexternal user can obtain identification results almost at the same timethat the medication is delivered into the chamber.

Furthermore, in order to verify that the present system decrease thepossibility of men-made errors during medication dispensing, anothercomparison was conducted. For example, Lorazepam, an anti-anxiety drug,is the most misidentified drug due to high appearance similarities withother drugs when being dispensed. Firstly, the RTI of Lorazepam wasinputted into the present system to compare with all medications in thelibrary. After computing, learning model proposed several candidatemediations which might be possibly misidentified based on appearancesimilarity analyzed by the learning model. The candidate lists can helpclinical stuffs double check whether the chosen medication is correct.Therefore, by the medication management system of the presentdisclosure, the accuracy for drug dispensing can be improved and thehuman errors can be minimized for ensuring patient safety.

To sum up, the method and the system for medication management accordingto the present disclosure can achieve the forgoing object by applyingimage processing steps to combine images into one and deep learningmodel in a real-time manner. The advantages of the present disclosureare not only efficiently processing raw images regardless of theorientation of objects (i.e., medications), but also increasing theaccuracy for medication classification via their appearances.

It will be understood that the above description of embodiments is givenby way of example only and that various modifications may be made bythose with ordinary skill in the art. The above specification, examplesand data provide a complete description of the structure and use ofexemplary embodiments of the invention. Although various embodiments ofthe invention have been described above with a certain degree ofparticularity, or with reference to one or more individual embodiments,those with ordinary skill in the art could make numerous alterations tothe disclosed embodiments without departing from the spirit or scope ofthis invention.

What is claimed is:
 1. A computer implemented method for building amedication library, comprising, (a) receiving a plurality of raw imagesof a blister package of a medication; (b) juxtaposing two of theplurality of raw images to produce a juxtaposed image, in which the tworaw images are different from each other; (c) processing the juxtaposedimage to produce a reference image; and (d) establishing the medicationlibrary with the aid of the reference image, wherein the step (b)comprises, (b-1) respectively processing the plurality of raw images toproduce a plurality of first processed images respectively havingdefined contours, (b-2) identifying the corners of each defined contoursof the first processed images to determine the coordinates thereof;(b-3) rotating each of the first processed images of the step (b-1)based on the determined coordinates of the step (b-2) to produce aplurality of second processed images; and (b-4) juxtaposing any two ofthe second processed images to produce the juxtaposed image of the step(b); and wherein the juxtaposed image produced in the step (b) fits afixed template.
 2. The computer implemented method of claim 1, furthercomprising capturing the plurality of raw images of the blister packageof the medication simultaneously prior to the step (a).
 3. The computerimplemented method of claim 1, wherein each of the plurality of rawimages of step (b-1) are subjected to treatments of, (i) agrayscale-converting treatment, (ii) a noise-reduction treatment, (iii)an edge-identification treatment, (iv) a convex hull treatment, and (v)a contouring treatment.
 4. The computer implemented method of claim 1,wherein the step (b-2) is carried out by a line transforming algorithmor a centroid algorithm.
 5. The computer implemented method of claim 1,wherein the juxtaposed image comprises the image of both sides of theblister package of the medication.
 6. The computer implemented method ofclaim 1, wherein the step (c) is carried out by a machine learningalgorithm.
 7. A computer implemented method for identifying a medicationvia its blister package, comprising, (a) obtaining the front and backimages of the blister package of the medication simultaneously; (b)juxtaposing the front and back images of the step (a) to produce acandidate image that fits the fixed template; (c) comparing thecandidate image with a reference image of a medication libraryestablished by the method of claim 1; and (d) outputting the result ofthe step (c).
 8. The computer implemented method of claim 7, wherein thestep comprises, (b-1) respectively processing the front and back imagesof the step (a) to produce two first processed images respectivelyhaving defined contours; (b-2) identifying the corners of each definedcontours of the two first processed images to determine the coordinatesthereof; (b-3) rotating each of the two first processed images of thestep (b-1) based on the determined coordinates of the step (b-2) toproduce two second processed images; and (b-4) combining the two secondprocessed images of the step (b-3) to produce the candidate image of thestep (b).
 9. The computer implemented method of claim 8, wherein thefront and back images of the step (b-1) are respectively subjected totreatments of, (i) a grayscale-converting treatment, (ii) anoise-reduction treatment, (iii) an edge-identification treatment, (iv)a convex hull treatment, and (v) a contouring treatment.
 10. Thecomputer implemented method of claim 8, wherein the step (b-2) iscarried out by a line transforming algorithm or a centroid algorithm.11. The computer implemented method of claim 7, wherein the step (c) iscarried out by a machine learning algorithm.
 12. The computerimplemented method of claim 7, further comprising transferring thecandidate image into the medication library prior to the step (c).
 13. Apharmaceutical management system, comprising, an image capturing deviceconfigured to capture a plurality of images of a package of amedication; an image processor programmed with instructions to execute amethod for producing a candidate image, wherein the method comprises,(1) respectively processing the plurality of images of the package ofthe medication to produce a plurality of first processed imagesrespectively having defined contours; (2) identifying the corners ofeach defined contours of the first processed images to determine thecoordinates thereof; (3) rotating each of the first processed imagesbased on the identified coordinates of the step (2) to produce aplurality of second processed images; and (4) juxtaposing two of thesecond processed images of the step (3) to produce the candidate image,in which the two second processed images are different from each other;and a machine learning processor programmed with instructions to executea method for comparing the candidate image with a reference image of amedication library established by the method of claim
 1. 14. The systemof claim 13, wherein the image capturing device comprises, a transparentboard on which the medication is placed; and two image-capturing unitsindividually disposed above each side of the transparent board.
 15. Thesystem of claim 13, wherein each of the plurality of images of step (1)are subjected to treatments of, (i) a grayscale-converting treatment,(ii) a noise-reduction treatment, (iii) an edge-identificationtreatment, (iv) a convex hull treatment, and (v) a contouring treatment.16. The system of claim 13, wherein the step (2) is carried out by aline transforming algorithm or a centroid algorithm.
 17. The system ofclaim 13, wherein the method for comparing the candidate image with areference image of a medication library is carried out by a machinelearning algorithm.